Dual aperture zoom digital camera

ABSTRACT

A dual-aperture zoom digital camera operable in both still and video modes. The camera includes Wide and Tele imaging sections with respective lens/sensor combinations and image signal processors and a camera controller operatively coupled to the Wide and Tele imaging sections. The Wide and Tele imaging sections provide respective image data. The controller is configured to combine in still mode at least some of the Wide and Tele image data to provide a fused output image from a particular point of view, and to provide without fusion continuous zoom video mode output images, each output image having a given output resolution, wherein the video mode output images are provided with a smooth transition when switching between a lower zoom factor (ZF) value and a higher ZF value or vice versa, and wherein at the lower ZF the output resolution is determined by the Wide sensor while at the higher ZF value the output resolution is determined by the Tele sensor.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. patentapplication Ser. No. 16/198,181 filed Nov. 21, 2018, which was acontinuation application of U.S. patent application Ser. No. 16/048,242filed Jul. 28, 2018 (issued as U.S. Pat. No. 10,225,479), which was acontinuation application of U.S. patent application Ser. No. 15/865,869,filed Jan. 9, 2018, which was a continuation application of U.S. patentapplication Ser. No. 15/424,853 filed Feb. 5, 2017 (issued as U.S. Pat.No. 10,015,408), which was a continuation application of U.S. patentapplication Ser. No. 14/880,251 filed Oct. 11, 2015 (issued as U.S. Pat.No. 9,661,233), which was a Continuation application of U.S. patentapplication Ser. No. 14/365,711 filed Jun, 16, 2014 (issued as U.S. Pat.No. 9,185,291), which was a 371 application from international patentapplication PCT/IB2014/062180 filed Jun. 12, 2014, and is related to andclaims priority from U.S. Provisional Patent Application No. 61/834,486having the same title and filed Jun. 13, 2013, which is incorporatedherein by reference in its entirety.

FIELD

Embodiments disclosed herein relate in general to digital cameras and inparticular to thin zoom digital cameras with both still image and videocapabilities

BACKGROUND

Digital camera modules are currently being incorporated into a varietyof host devices. Such host devices include cellular telephones, personaldata assistants (PDAs), computers, and so forth. Consumer demand fordigital camera modules in host devices continues to grow.

Host device manufacturers prefer digital camera modules to be small, sothat they can be incorporated into the host device without increasingits overall size. Further, there is an increasing demand for suchcameras to have higher-performance characteristics. One suchcharacteristic possessed by many higher-performance cameras (e.g.,standalone digital still cameras) is the ability to vary the focallength of the camera to increase and decrease the magnification of theimage. This ability, typically accomplished with a zoom lens, is knownas optical zooming. “Zoom” is commonly understood as a capability toprovide different magnifications of the same scene and/or object bychanging the focal length of an optical system, with a higher level ofzoom associated with greater magnification and a lower level of zoomassociated with lower magnification. Optical zooming is typicallyaccomplished by mechanically moving lens elements relative to eachother. Such zoom lenses are typically more expensive, larger and lessreliable than fixed focal length lenses. An alternative approach forapproximating the zoom effect is achieved with what is known as digitalzooming. With digital zooming, instead of varying the focal length ofthe lens, a processor in the camera crops the image and interpolatesbetween the pixels of the captured image to create a magnified butlower-resolution image.

Attempts to use multi-aperture imaging systems to approximate the effectof a zoom lens are known. A multi-aperture imaging system (implementedfor example in a digital camera) includes a plurality of opticalsub-systems (also referred to as “sub-cameras”). Each sub-cameraincludes one or more lenses and/or other optical elements which definean aperture such that received electro-magnetic radiation is imaged bythe optical sub-system and a resulting image is directed towards atwo-dimensional (2D) pixelated image sensor region. The image sensor (orsimply “sensor”) region is configured to receive the image and togenerate a set of image data based on the image. The digital camera maybe aligned to receive electromagnetic radiation associated with sceneryhaving a given set of one or more objects. The set of image data may berepresented as digital image data, as well known in the art. Hereinafterin this description, “image” “image data” and “digital image data” maybe used interchangeably. Also, “object” and “scene” may be usedinterchangeably.

Multi-aperture imaging systems and associated methods are described forexample in US Patent Publications No. 2008/0030592, 2010/0277619 and2011/0064327. In US 2008/0030592, two sensors are operatedsimultaneously to capture an image imaged through an associated lens. Asensor and its associated lens form a lens/sensor combination. The twolenses have different focal lengths. Thus, even though each lens/sensorcombination is aligned to look in the same direction, each captures animage of the same subject but with two different fields of view (FOVs).One sensor is commonly called “Wide” and the other “Tele”. Each sensorprovides a separate image, referred to respectively as “Wide” (or “W”)and “Tele” (or “T”) images. A W-image reflects a wider FOV and has lowerresolution than the T-image. The images are then stitched (fused)together to form a composite (“fused”) image. In the composite image,the central portion is formed by the relatively higher-resolution imagetaken by the lens/sensor combination with the longer focal length, andthe peripheral portion is formed by a peripheral portion of therelatively lower-resolution image taken by the lens/sensor combinationwith the shorter focal length. The user selects a desired amount of zoomand the composite image is used to interpolate values from the chosenamount of zoom to provide a respective zoom image. The solution offeredby US 2008/0030592 requires, in video mode, very large processingresources in addition to high frame rate requirements and high powerconsumption (since both cameras are fully operational).

US 2010/0277619 teaches a camera with two lens/sensor combinations, thetwo lenses having different focal lengths, so that the image from one ofthe combinations has a FOV approximately 2-3 times greater than theimage from the other combination. As a user of the camera requests agiven amount of zoom, the zoomed image is provided from the lens/sensorcombination having a FOV that is next larger than the requested FOV.Thus, if the requested FOV is less than the smaller FOV combination, thezoomed image is created from the image captured by that combination,using cropping and interpolation if necessary. Similarly, if therequested FOV is greater than the smaller FOV combination, the zoomedimage is created from the image captured by the other combination, usingcropping and interpolation if necessary. The solution offered by US2010/0277619 leads to parallax artifacts when moving to the Tele camerain video mode.

In both US 2008/0030592 and US 2010/0277619, different focal lengthsystems cause Tele and Wide matching FOVs to be exposed at differenttimes using CMOS sensors. This degrades the overall image quality.Different optical F numbers (“F#”) cause image intensity differences.Working with such a dual sensor system requires double bandwidthsupport, i.e. additional wires from the sensors to the following HWcomponent. Neither US 2008/0030592 nor US 2010/0277619 deal withregistration errors. Neither US2008/000592 nor US 2010/0277619 refer topartial fusion, i.e. fusion of less than all the pixels of both Wide andTele images in still mode.

US 2011/0064327 discloses multi-aperture imaging systems and methods forimage data fusion that include providing first and second sets of imagedata corresponding to an imaged first and second scene respectively. Thescenes overlap at least partially in an overlap region, defining a firstcollection of overlap image data as part of the first set of image data,and a second collection of overlap image data as part of the second setof image data. The second collection of overlap image data isrepresented as a plurality of image data sub-cameras such that each ofthe sub-cameras is based on at least one characteristic of the secondcollection, and each sub-camera spans the overlap region. A fused set ofimage data is produced by an image processor, by modifying the firstcollection of overlap image data based on at least a selected one of,but less than all of, the image data sub-cameras. The systems andmethods disclosed in this application deal solely with fused stillimages.

None of the known art references provide a thin (e.g. fitting in acell-phone) dual-aperture zoom digital camera with fixed focal lengthlenses, the camera configured to operate in both still mode and videomode to provide still and video images, wherein the camera configurationuses partial or full fusion to provide a fused image in still mode anddoes not use any fusion to provide a continuous, smooth zoom in videomode.

Therefore there is a need for, and it would be advantageous to have thindigital cameras with optical zoom operating in both video and still modethat do not suffer from commonly encountered problems and disadvantages,some of which are listed above.

SUMMARY

Embodiments disclosed herein teach the use of dual-aperture (alsoreferred to as dual-lens or two-sensor) optical zoom digital cameras.The cameras include two sub-cameras, a Wide sub-camera and a Telesub-camera, each sub-camera including a fixed focal length lens, animage sensor and an image signal processor (ISP). The Tele sub-camera isthe higher zoom sub-camera and the Wide sub-camera is the lower zoomsub-camera. In some embodiments, the lenses are thin lenses with shortoptical paths of less than about 9 mm. In some embodiments, thethickness/effective focal length (EFL) ratio of the Tele lens is smallerthan about 1. The image sensor may include two separate 2D pixelatedsensors or a single pixelated sensor divided into at least two areas.The digital camera can be operated in both still and video modes. Instill mode, zoom is achieved “with fusion” (full or partial), by fusingW and T images, with the resulting fused image including alwaysinformation from both W and T images. Partial fusion may be achieved bynot using fusion in image areas where the Tele image is not focused.This advantageously reduces computational requirements (e.g. time).

In video mode, optical zoom is achieved “without fusion”, by switchingbetween the W and T images to shorten computational time requirements,thus enabling high video rate. To avoid discontinuities in video mode,the switching includes applying additional processing blocks, whichinclude image scaling and shifting.

In order to reach optical zoom capabilities, a different magnificationimage of the same scene is captured (grabbed) by each camera sub-camera,resulting in FOV overlap between the two sub-cameras. Processing isapplied on the two images to fuse and output one fused image in stillmode. The fused image is processed according to a user zoom factorrequest. As part of the fusion procedure, up-sampling may be applied onone or both of the grabbed images to scale it to the image grabbed bythe Tele sub-camera or to a scale defined by the user. The fusion orup-sampling may be applied to only some of the pixels of a sensor.Down-sampling can be performed as well if the output resolution issmaller than the sensor resolution.

The cameras and associated methods disclosed herein address and correctmany of the problems and disadvantages of known dual-aperture opticalzoom digital cameras. They provide an overall zoom solution that refersto all aspects: optics, algorithmic processing and system hardware (HW).The proposed solution distinguishes between video and still mode in theprocessing flow and specifies the optical requirements and HWrequirements. In addition, it provides an innovative optical design thatenables a low TTL/EFL ratio using a specific lens curvature order.

Due to the large focal length, objects that are in front or behind theplane of focus appear very blurry, and a nice foreground-to-backgroundcontrast is achieved. However, it is difficult to create such a blurusing a compact camera with a relatively short focal length and smallaperture size, such as a cell-phone camera. In some embodiments, adual-aperture zoom system disclosed herein can be used to capture ashallow DOF photo (shallow compared with a DOF of a Wide camera alone),by taking advantage of the longer focal length of the Tele lens. Thereduced DOF effect provided by the longer Tele focal length can befurther enhanced in the final image by fusing data from an imagecaptured simultaneously with the

Wide lens. Depending on the distance to the object, with the Tele lensfocused on a subject of the photo, the Wide lens can be focused to acloser distance than the subject so that objects behind the subjectappear very blurry. Once the two images are captured, information fromthe out-of-focus blurred background in the Wide image is fused with theoriginal Tele image background information, providing a blurrierbackground and even shallower DOF.

In an embodiment there is provided a zoom digital camera comprising aWide imaging section that includes a fixed focal length Wide lens with aWide FOV, a Wide sensor and a Wide image signal processor (ISP), theWide imaging section operative to provide Wide image data of an objector scene; a Tele imaging section that includes a fixed focal length Telelens with a Tele FOV that is narrower than the Wide FOV, a Tele sensorand a Tele ISP, the Tele imaging section operative to provide Tele imagedata of the object or scene; and a camera controller operatively coupledto the Wide and Tele imaging sections, the camera controller configuredto combine in still mode at least some of the Wide and Tele image datato provide a fused output image of the object or scene from a particularpoint of view (POV), and to provide without fusion continuous zoom videomode output images of the object or scene, a camera controlleroperatively coupled to the Wide and Tele imaging sections, the cameracontroller configured to combine in still mode at least some of the Wideand Tele image data to provide a fused output image of the object orscene from a particular point of view and to provide without fusioncontinuous zoom video mode output images of the object or scene, eachoutput image having a respective output resolution, wherein the videooutput images are provided with a smooth transition when switchingbetween a lower zoom factor (ZF) value and a higher ZF value or viceversa, wherein at the lower ZF value the output resolution is determinedby the Wide sensor, and wherein at the higher ZF value the outputresolution is determined by the Tele sensor.

In an embodiment, the camera controller configuration to provide videooutput images with a smooth transition when switching between a lower ZFvalue and a higher ZF value or vice versa includes a configuration thatuses at high ZF secondary information from the Wide camera and uses atlow ZF secondary information from the Tele camera. As used herein,“secondary information” refers to white balance gain, exposure time,analog gain and color correction matrix.

In a dual-aperture camera image plane, as seen by each sub-camera (andrespective image sensor), a given object will be shifted and havedifferent perspective (shape). This is referred to as point-of-view(POV). The system output image can have the shape and position of eithersub-camera image or the shape or position of a combination thereof. Ifthe output image retains the Wide image shape then it has the Wideperspective POV. If it retains the Wide camera position then it has theWide position POV. The same applies for Tele images position andperspective. As used in this description, the perspective POV may be ofthe Wide or Tele sub-cameras, while the position POV may shiftcontinuously between the Wide and Tele sub-cameras. In fused images, itis possible to register Tele image pixels to a matching pixel set withinthe Wide image pixels, in which case the output image will retain theWide POV (“Wide fusion”). Alternatively, it is possible to register Wideimage pixels to a matching pixel set within the Tele image pixels, inwhich case the output image will retain the Tele POV (“Tele fusion”). Itis also possible to perform the registration after either sub-cameraimage is shifted, in which case the output image will retain therespective Wide or Tele perspective POV.

In an embodiment there is provided a method for obtaining zoom images ofan object or scene in both still and video modes using a digital camera,the method comprising the steps of providing in the digital camera aWide imaging section having a Wide lens with a Wide FOV, a Wide sensorand a Wide image signal processor (ISP), a Tele imaging section having aTele lens with a Tele FOV that is narrower than the Wide FOV, a Telesensor and a Tele ISP, and a camera controller operatively coupled tothe Wide and Tele imaging sections; and configuring the cameracontroller to combine in still mode at least some of the Wide and Teleimage data to provide a fused output image of the object or scene from aparticular point of view, and to provide without fusion continuous zoomvideo mode output images of the object or scene, each output imagehaving a respective output resolution, wherein the video mode outputimages are provided with a smooth transition when switching between alower ZF value and a higher ZF value or vice versa, and wherein at thelower ZF value the output resolution is determined by the Wide sensorwhile at the higher ZF value the output resolution is determined by theTele sensor.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting examples of embodiments disclosed herein are describedbelow with reference to figures attached hereto that are listedfollowing this paragraph. The drawings and descriptions are meant toilluminate and clarify embodiments disclosed herein, and should not beconsidered limiting in any way.

FIG. 1A shows schematically a block diagram illustrating a dual-aperturezoom imaging system disclosed herein;

FIG. 1B is a schematic mechanical diagram of the dual-aperture zoomimaging system of FIG. 1A:

FIG. 2 shows an example of Wide sensor, Tele sensor and their respectiveFOVs;

FIG. 3 shows a schematically embodiment of CMOS sensor image grabbingvs. time;

FIG. 4 shows schematically a sensor time configuration which enablessharing one sensor interface using dual sensor zoom system;

FIG. 5 shows an embodiment of a method disclosed herein for acquiring azoom image in capture mode;

FIG. 6 shows an embodiment of a method disclosed herein for acquiring azoom image in video/preview mode;

FIG. 7 shows a graph illustrating an effective resolution zoom factor;

FIG. 8 shows one embodiment of a lens block in a thin camera disclosedherein;

FIG. 9 shows another embodiment of a lens block in a thin cameradisclosed herein.

DETAILED DESCRIPTION

FIG. 1A shows schematically a block diagram illustrating an embodimentof a dual-aperture zoom imaging system (also referred to simply as“digital camera” or “camera”) disclosed herein and numbered 100. Camera100 comprises a Wide imaging section (“sub-camera”) that includes a Widelens block 102, a Wide image sensor 104 and a Wide image processor 106.Camera 100 further comprises a Tele imaging section (“sub-camera”) thatincludes a Tele lens block 108, a Tele image sensor 110 and a Tele imageprocessor 112. The image sensors may be physically separate or may bepart of a single larger image sensor. The Wide sensor pixel size can beequal to or different from the Tele sensor pixel size. Camera 100further comprises a camera fusion processing core (also referred to as“controller”) 114 that includes a sensor control module 116, a usercontrol module 118, a video processing module 126 and a captureprocessing module 128, all operationally coupled to sensor control block110. User control module 118 comprises an operational mode function 120,a region of interest (ROI) function 122 and a zoom factor (ZF) function124.

Sensor control module 116 is connected to the two sub-cameras and to theuser control module 118 and used to choose, according to the zoomfactor, which of the sensors is operational and to control the exposuremechanism and the sensor readout. Mode choice function 120 is used forchoosing capture/video modes. ROI function 122 is used to choose aregion of interest. As used herein, “ROI” is a user defined as asub-region of the image that may be exemplarily 4% or less of the imagearea. The ROI is the region on which both sub-cameras are focused on.Zoom factor function 124 is used to choose a zoom factor. Videoprocessing module 126 is connected to mode choice function 120 and usedfor video processing. Still processing module 128 is connected to themode choice function 120 and used for high image quality still modeimages. The video processing module is applied when the user desires toshoot in video mode. The capture processing module is applied when theuser wishes to shoot still pictures.

FIG. 1B is a schematic mechanical diagram of the dual-aperture zoomimaging system of FIG. 1A. Exemplary dimensions: Wide lens TTL=4.2 mmand EFL=3.5 mm; Tele lens TTL=6 mm and EFL=7 mm; both Wide and Telesensors ⅓ inch. External dimensions of Wide and Tele cameras: width (w)and length (l)=8.5 mm and height (h)=6.8 mm. Distance “d” between cameracenters=10 mm.

Following is a detailed description and examples of different methods ofuse of camera 100.

Design for Continuous and Smooth Zoom in Video Mode

In an embodiment, in order to reach high quality continuous and smoothoptical zooming in video camera mode while reaching real optical zoomusing fixed focal length sub-cameras, the system is designed accordingto the following rules (Equations 1-3):

Tan (FOV_(Wide))/Tan (FOV_(Tele))=PL _(Wide) / PL _(video)   (1)

where Tan refers to “tangent”, while FOV_(Wide) and FOV_(Tele) referrespectively to the Wide and Tele lens fields of view (in degrees). Asused herein, the FOV is measured from the center axis to the corner ofthe sensor (i.e. half the angle of the normal definition). PL_(Wide) andPL_(video) refer respectively to the “in-line” (i.e. in a line) numberof Wide sensor pixels and in-line number of output video format pixels.The ratio PL_(Wide)/PL_(video) is called an “oversampling ratio”. Forexample, in order to get full and continuous optical zoom experiencewith a 12 Mp sensor (sensor dimensions 4000×3000) and a required 1080p(dimension 1920×1080) video format, the FOV ratio should be4000/1920=2.083. Moreover, if the Wide lens FOV is given asFOV_(Wide)=37.5°, the required Tele lens FOV is 20.2° The zoom switchingpoint is set according to the ratio between sensor pixels in-line andthe number of pixels in-line in the video format and defined as:

Z _(switch) =PL _(Wide) / PL _(video)   (2)

Maximum optical zoom is reached according to the following formula:

Z _(max)=Tan (FOV_(Wide))/Tan (FOV_(Tele))*PL _(Tele) /PL _(video)   (3)

For example: for the configuration defined above and assumingPL_(Tele)=4000 and PL_(video)=1920, Z_(max)=4.35.

In an embodiment, the sensor control module has a setting that dependson the Wide and Tele FOVs and on a sensor oversampling ratio, thesetting used in the configuration of each sensor. For example, whenusing a 4000×3000 sensor and when outputting a 1920×1080 image, theoversampling ratio is 4000/1920=2.0833.

In an embodiment, the Wide and Tele FOVs and the oversampling ratiosatisfy the condition

0.8*PL _(Wide) /PL _(video)<Tan (FOV_(Wide))/Tan (FOV_(Tele))<1.2*PL_(Wide) /PL _(video).   (4)

Still Mode Operation/Function

In still camera mode, the obtained image is fused from informationobtained by both sub-cameras at all zoom levels, see FIG. 2, which showsa Wide sensor 202 and a Tele sensor 204 and their respective FOVs.Exemplarily, as shown, the Tele sensor FOV is half the Wide sensor FOV.The still camera mode processing includes two stages: (1) setting HWsettings and configuration, where a first objective is to control thesensors in such a way that matching FOVs in both images (Tele and Wide)are scanned at the same time. A second objective is to control therelative exposures according to the lens properties. A third objectiveis to minimize the required bandwidth from both sensors for the ISPs;and (2) image processing that fuses the Wide and the Tele images toachieve optical zoom, improves SNR and provides wide dynamic range.

FIG. 3 shows image line numbers vs. time for an image section capturedby CMOS sensors. A fused image is obtained by line (row) scans of eachimage. To prevent matching FOVs in both sensors to be scanned atdifferent times, a particular configuration is applied by the cameracontroller on both image sensors while keeping the same frame rate. Thedifference in FOV between the sensors determines the relationshipbetween the rolling shutter time and the vertical blanking time for eachsensor. In the particular configuration, the scanning is synchronizedsuch that the same points of the object in each view are obtainedsimultaneously.

Specifically with reference to FIG. 3 and according to an embodiment ofa method disclosed herein, the configuration to synchronize the scanningincludes: setting the Tele sensor vertical blanking time VB_(Tele) toequal the Wide sensor vertical blanking time VB_(Wide) plus half theWide sensor rolling shutter time RST_(Wide); setting the Tele and Widesensor exposure times ET_(Tele) and ET_(Wide) to be equal or different;setting the Tele sensor rolling shutter time RST_(Tele) to be0.5*RST_(Wide); and setting the frame rates of the two sensors to beequal. This procedure results in identical image pixels in the Tele andWide sensor images being exposed at the same time

In another embodiment, the camera controller synchronizes the Wide andTele sensors so that for both sensors the rolling shutter starts at thesame time.

The exposure times applied to the two sensors could be different, forexample in order to reach same image intensity using different F# anddifferent pixel size for the Tele and Wide systems. In this case, therelative exposure time may be configured according to the formula below:

ET_(Tele)=ET_(Wide)·(F# _(Tele) /F# _(Wide))²·(Pixel size_(Wide)/Pixelsize_(Tele))   (5)

Other exposure time ratios may be applied to achieve wide dynamic rangeand improved SNR. Fusing two images with different intensities willresult in wide dynamic range image.

In more detail with reference to FIG. 3, in the first stage, after theuser chooses a required zoom factor ZF, the sensor control moduleconfigures each sensor as follows:

1) Cropping index Wide sensor:

Y _(Wide start)=½·PC_(Wide)(1−1/ZF)

Y _(Wide end)=½PC_(Wide)(1+1/ZF)

where PC is the number of pixels in a column, and Y is the row number

2) Cropping index Tele sensor:

If ZF>Tan (FOV_(Wide))/Tan (FOV_(Tele)), then

Y _(Tele star t)=½·PC_(Tele)(1−(1/ZF)·Tan (FOV_(Tele))/Tan (FOV_(Wide)))

Y _(Tele end)=½·PC_(Tele)(1+(1/ZF)·Tan (FOV_(Tele))/Tan (FOV_(wide)))

If ZF<Tan (FOV_(Wide))/Tan (FOV_(Tele)), then

Y_(Tele start)−=0

Y_(Tele end)=PC_(Tele)

This will result in an exposure start time of the Tele sensor with adelay of (in numbers of lines, relative to the Wide sensor start time):

(1−ZF/((Tan (FOV_(Wide))/Tan (FOV_(Tele))))·1/(2·FPS)   (6)

where FPS is the sensor's frame per second configuration. In cases whereZF> Tan (FOV_(Wide))/Tan (FOV_(Tele)), no delay will be introducedbetween Tele and Wide exposure starting point. For example, for a casewhere Tan (FOV_(Wide))/Tan (FOV_(Tele))=2 and ZF=1, the Tele image firstpixel is exposed ¼·(1/FPS) second after the Wide image first pixel wasexposed.

After applying the cropping according to the required zoom factor, thesensor rolling shutter time and the vertical blank should be configuredin order to satisfy the equation to keep the same frame rate:

VB_(Wide)+RST_(Wide)=VB_(Tele)+RST_(Tele)   (7)

FIG. 3 exemplifies Eq. (7), One way to satisfy Eq. (7) is to increasethe RST_(Wide). Controlling the RST_(Wide) may be done by changing thehorizontal blanking (HB) of the Wide sensor. This will cause a delaybetween the data coming out from each row of the Wide sensor.

Generally, working with a dual-sensor system requires multiplying thebandwidth to the following block, for example the ISP. For example,using 12 Mp working at 30 fps, 10 bit per pixel requires working at 3.6Gbit/sec. In this example, supporting this bandwidth requires 4 lanesfrom each sensor to the respective following ISP in the processingchain. Therefore, working with two sensors requires double bandwidth(7.2 Gbit/sec) and 8 lanes connected to the respective following blocks.The bandwidth can be reduced by configuring and synchronizing the twosensors. Consequently, the number of lanes can be half that of aconventional configuration (3.6 Gbit/sec).

FIG. 4 shows schematically a sensor time configuration that enablessharing one sensor interface using a dual-sensor zoom system, whilefulfilling the conditions in the description of FIG. 3 above. Forsimplicity, assuming the Tele sensor image is magnified by a factor of 2compared with the Wide sensor image, the Wide sensor horizontal blankingtime HB_(Wide) is set to twice the Wide sensor line readout time. Thiscauses a delay between output Wide lines. This delay time matchesexactly the time needed to output two lines from the Tele sensor. Afteroutputting two lines from the Tele sensor, the Tele sensor horizontalblanking time H_(Wide) is set to be one Wide line readout time, so,while the Wide sensor outputs a row from the sensor, no data is beingoutput from the Tele sensor. For this example, every 3^(rd) line in theTele sensor is delayed by an additional HB_(Tele). In this delay time,one line from the Wide sensor is output from the dual-sensor system.After the sensor configuration stage, the data is sent in parallel or byusing multiplexing into the processing section.

FIG. 5 shows an embodiment of a method disclosed herein for acquiring azoom image in still mode. In ISP step 502, the data of each sensor istransferred to the respective ISP component, which performs on the datavarious processes such as denoising, demosaicing, sharpening, scaling,etc, as known in the art. After the processing in step 502, allfollowing actions are performed in capture processing core 128: inrectification step 504, both Wide and Tele images are aligned to be onthe epipolar line; in registration step 506, mapping between the Wideand the Tele aligned images is performed to produce a registration map;in resampling step 508, the Tele image is resampled according to theregistration map, resulting in a re-sampled Tele image; in decision step510, the re-sampled Tele image and the Wide image are processed todetect errors in the registration and to provide a decision output. Inmore detail, in step 510, the re-sampled Tele image data is comparedwith the Wide image data and if the comparison detects significantdissimilarities, an error is indicated. In this case, the Wide pixelvalues are chosen to be used in the output image. Then, in fusion step512, the decision output, re-sampled Tele image and the Wide image arefused into a single zoom image.

To reduce processing time and power, steps 506, 508, 510, 512 could bebypassed by not fusing the images in non-focused areas. In this case,all steps specified above should be applied on focused areas only. Sincethe Tele optical system will introduce shallower depth of field than theWide optical system, defocused areas will suffer from lower contrast inthe Tele system.

Zoom-In and Zoom-Out in Still Camera Mode

We define the following: TFOV=tan (camera FOV/2). “Low ZF” refers to allZF that comply with ZF<Wide TFOV/Tele TFOV. “High ZF” refers to all ZFthat comply with ZF>Wide TFOV/Tele TFOV. “ZFT” refers to a ZF thatcomplies with ZF=Wide TFOV/Tele TFOV. In one embodiment, zoom-in andzoom-out in still mode is performed as follows:

Zoom-in: at low ZF up to slightly above ZFT, the output image is adigitally zoomed, Wide fusion output. For the up-transfer ZF, the Teleimage is shifted and corrected by global registration (GR) to achievesmooth transition. Then, the output is transformed to a Tele fusionoutput. For higher (than the up-transfer) ZF, the output is the Telefusion output digitally zoomed.

Zoom-out: at high ZF down to slightly below ZFT, the output image is adigitally zoomed, Tele fusion output. For the down-transfer ZF, the Wideimage is shifted and corrected by GR to achieve smooth transition. Then,the output is transformed to a Wide fusion output. For lower (than thedown-transfer) ZF, the output is basically the down-transfer ZF outputdigitally zoomed but with gradually smaller Wide shift correction, untilfor ZF=1 the output is the unchanged Wide camera output.

In another embodiment, zoom-in and zoom-out in still mode is performedas follows:

Zoom-in: at low ZF up to slightly above ZFT, the output image is adigitally zoomed, Wide fusion output. For the up-transfer ZF and above,the output image is the Tele fusion output.

Zoom-out: at high ZF down to slightly below ZFT, the output image is adigitally zoomed, Tele fusion output. For the down-transfer ZF andbelow, the output image is the Wide fusion output.

Video Mode Operation/Function Smooth Transition

When a dual-aperture camera switches the camera output betweensub-cameras or points of view, a user will normally see a “jump”(discontinuous) image change. However, a change in the zoom factor forthe same camera and POV is viewed as a continuous change. A “smoothtransition” is a transition between cameras or POVs that minimizes thejump effect. This may include matching the position, scale, brightnessand color of the output image before and after the transition. However,an entire image position matching between the sub-camera outputs is inmany cases impossible, because parallax causes the position shift to bedependent on the object distance. Therefore, in a smooth transition asdisclosed herein, the position matching is achieved only in the ROIregion while scale brightness and color are matched for the entireoutput image area.

Zoom-In and Zoom-Out in Video Mode

In video mode, sensor oversampling is used to enable continuous andsmooth zoom experience. Processing is applied to eliminate the changesin the image during crossover from one sub-camera to the other. Zoomfrom 1 to Z_(switch) is performed using the Wide sensor only. FromZ_(switch) and on, it is performed mainly by the Tele sensor. To prevent“jumps” (roughness in the image), switching to the Tele image is doneusing a zoom factor which is a bit higher (Z_(switch)+/ΔVoom) thanZ_(switch). ΔZoom is determined according to the system's properties andis different for cases where zoom-in is applied and cases where zoom-outis applied (ΔZoom_(in)≠ΔZoom_(out)). This is done to prevent residualjumps artifacts to be visible at a certain zoom factor. The switchingbetween sensors, for an increasing zoom and for decreasing zoom, is doneon a different zoom factor.

The zoom video mode operation includes two stages: (1) sensor controland configuration, and (2) image processing. In the range from 1 toZ_(switch), only the Wide sensor is operational, hence, power can besupplied only to this sensor. Similar conditions hold for a Wide AFmechanism. From Z_(switch)+ΔZoom to Z_(max) only the Tele sensor isoperational, hence, power is supplied only to this sensor. Similarly,only the Tele sensor is operational and power is supplied only to it fora Tele AF mechanism. Another option is that the Tele sensor isoperational and the Wide sensor is working in low frame rate. FromZ_(switch) to Z_(switch)+ΔZoom, both sensors are operational.

Zoom-in: at low ZF up to slightly above ZFT, the output image is thedigitally zoomed, unchanged Wide camera output. For the up-transfer ZF,the output is a transformed Tele sub-camera output, where thetransformation is performed by a global registration (GR) algorithm toachieve smooth transition. For higher (than the up-transfer), the outputis the transfer ZF output digitally zoomed.

Zoom-out: at high ZF down to slightly below ZFT, the output image is thedigitally zoomed transformed Tele camera output. For the down-transferZF, the output is a shifted

Wide camera output, where the Wide shift correction is performed by theGR algorithm to achieve smooth transition, i.e. with no jump in the ROIregion. For lower (than the down-transfer) ZF, the output is basicallythe down-transfer ZF output digitally zoomed but with gradually smallerWide shift correction, until for ZF=1 the output is the unchanged Widecamera output.

FIG. 6 shows an embodiment of a method disclosed herein for acquiring azoom image in video/preview mode for 3 different zoom factor (ZF)ranges: (a) ZF range=1: Z_(switch); (b) ZF range=Z_(switch) : Z_(switch)+ΔZoom_(in): and (c) Zoom factor range=Z_(switch)+ΔZoom_(in) : Z_(max).The description is with reference to a graph of effective resolution vs.zoom value (FIG. 7). In step 602, sensor control module 116 chooses(directs) the sensor (Wide, Tele or both) to be operational.Specifically, if the ZF range=1:Z_(switch), module 116 directs the Widesensor to be operational and the Tele sensor to be non-operational. Ifthe ZF range is Z_(switch) : Z_(switch)ΔZoom_(in), module 116 directsboth sensors to be operational and the zoom image is generated from theWide sensor. If the ZF range is Z_(switch)+ΔZoom_(th) : Z_(max), module116 directs the Wide sensor to be non-operational and the Tele sensor tobe operational. After the sensor choice in step 602, all followingactions are performed in video processing core 126. Optionally, in step604, color balance is calculated if two images are provided by the twosensors. Optionally yet, in step 606, the calculated color balance isapplied in one of the images (depending on the zoom factor). Furtheroptionally, in step 608, registration is performed between the Wide andTele images to output a transformation coefficient. The transformationcoefficient can be used to set an AF position in step 610. In step 612,an output of any of steps 602-608 is applied on one of the images(depending on the zoom factor) for image signal processing that mayinclude denoising, demosaicing, sharpening, scaling, etc. In step 614,the processed image is resampled according to the transformationcoefficient, the requested ZF (obtained from zoom function 124) and theoutput video resolution (for example 1080p). To avoid a transition pointto be executed at the same ZF, ΔZoom can change while zooming in andwhile zooming out. This will result in hysteresis in the sensorswitching point.

In more detail, for ZF range 1 : Z_(switch), for ZF<Z_(switch), the Wideimage data is transferred to the ISP in step 612 and resampled in step614. For ZF range=Z_(switch) : Z_(switch)+ΔZoom_(in), both sensors areoperational and the zoom image is generated from the Wide sensor. Thecolor balance is calculated for both images according to a given ROI. Inaddition, for a given ROI, registration is performed between the Wideand Tele images to output a transformation coefficient. Thetransformation coefficient is used to set an AF position. Thetransformation coefficient includes the translation between matchingpoints in the two images. This translation can be measured in a numberof pixels. Different translations will result in a different number ofpixel movements between matching points in the images. This movement canbe translated into depth and the depth can be translated into an AFposition. This enables to set the AF position by only analyzing twoimages (Wide & Tele). The result is fast focusing.

Both color balance ratios and transformation coefficient are used in theISP step. In parallel, the Wide image is processed to provide aprocessed image, followed by resampling. For ZFrange=Z_(switch)+ΔZoom_(in) : Z_(max) and for Zoomfactor>Z_(switch)+ΔZoom_(in), the color balance calculated previously isnow applied on the Tele image. The Tele image data is transferred to theISP in step 612 and resampled in step 614. To eliminate crossoverartifacts and to enable smooth transition to the Tele image, theprocessed Tele image is resampled according to the transformationcoefficient, the requested ZF (obtained from zoom function 124) and theoutput video resolution (for example 1080p).

FIG. 7 shows the effective resolution as a function of the zoom factorfor a zoom-in case and for a a zoom-out case ΔZoom_(up) is set when wezoom in, and ΔZoom_(down) is set when we zoom out. Setting ΔZoom_(up) tobe different from ΔZoom_(down) will result in transition between thesensors to be performed at different zoom factor (“hysteresis”) whenzoom-in is used and when zoom-out is used. This hysteresis phenomenon inthe video mode results in smooth continuous zoom experience.

Optical Design

Additional optical design considerations were taken into account toenable reaching optical zoom resolution using small total track length(TTL). These considerations refer to the Tele lens. In an embodiment,the camera is “thin” (see also FIG. 1B) in the sense that is has anoptical path of less than 9 mm and a thickness/focal length (FP) ratiosmaller than about 0.85. Exemplarily, as shown in FIG. 8, such a thincamera has a lens block that includes (along an optical axis startingfrom an object) five lenses: a first lens element 802 with positivepower and two lenses 804 and 806 and with negative power, a fourth lens808 with positive power and a fifth lens 810 with negative power. In theembodiment of FIG. 8, the EFL is 7 mm, the TTL is 4.7 mm, f=6.12 andFOV=20°. Thus the Tele lens TTL/EFL ratio is smaller than 0.9. In otherembodiments, the Tele lens TTL/EFL ratio may be smaller than 1.

In another embodiment of a lens block in a thin camera, shown in FIG. 9,the camera has a lens block that includes (along an optical axisstarting from an object) a first lens element 902 with positive power asecond lens element 904 with negative power, a third lens element withpositive power 906 and a fourth lens element with negative power 908,and a fifth lens element 910 with positive or negative power In thisembodiment, f=7.14, F#=3.5, TTL=5.8 mm and FOV=22.7°.

In conclusion, dual aperture optical zoom digital cameras and associatemethods disclosed herein reduce the amount of processing resources,lower frame rate requirements, reduce power consumption, remove parallaxartifacts and provide continuous focus (or provide loss of focus) whenchanging from Wide to Tele in video mode. They provide a dramaticreduction of the disparity range and avoid false registration in capturemode. They reduce image intensity differences and enable work with asingle sensor bandwidth instead of two, as in known cameras.

All patent applications mentioned in this specification are hereinincorporated in their entirety by reference into the specification, tothe same extent as if each individual patent application wasspecifically and individually indicated to be incorporated herein byreference. In addition, citation or identification of any reference inthis application shall not be construed as an admission that suchreference is available as prior art to the present disclosure.

While this disclosure has been described in terms of certain embodimentsand generally associated methods, alterations and permutations of theembodiments and methods will be apparent to those skilled in the art.The disclosure is to be understood as not limited by the specificembodiments described herein, but only by the scope of the appendedclaims.

What is claimed is:
 1. A multi-aperture imaging system, comprising: a) afirst camera that provides a first image, the first camera having afirst field of view (FOV1) and a first sensor with a first pixel size;and b) a second camera that provides a second image, the second camerahaving a second field of view (FOV2) such that FOV2<FOV1 and a secondsensor with a second pixel size different from the first pixel size,wherein the second camera has a total track length (TTL), an effectivefocal length (EFL) and a TTL/EFL ratio smaller than 0.9.
 2. Themulti-aperture imaging system of claim 1, wherein the second camera hasan optical path of less than 9 mm.
 3. The multi-aperture imaging systemof claim 2, wherein the second camera has a fixed focal length.
 4. Themulti-aperture imaging system of claim 3, wherein the first camera has afixed focal length.
 5. The multi-aperture imaging system of claim 2,wherein the multi-aperture imaging system has a still mode and a videomode.
 6. The multi-aperture imaging system of claim 1, furthercomprising a camera controller connected to the first camera and to thesecond camera, the camera controller configured to output a fused imageusing the first image and the second image.
 7. The multi-apertureimaging system of claim 6, wherein the fused image is generated byregistering pixels within the second sensor to pixels within the firstsensor.
 8. The multi-aperture imaging system of claim 6, wherein thefused image is generated by registering pixels within the first sensorto pixels within the second sensor.
 9. A multi-aperture imaging system,comprising: a) a first camera that provides a first image, the firstcamera having a first field of view (FOV1) and a first sensor with afirst pixel size; and b) a second camera that provides a second image,the second camera having a second field of view (FOV2) such thatFOV2<FOV1 and a second sensor having the first pixel size, wherein asecond camera has a total track length (TTL), an effective focal length(EFL), and a TTL/EFL ratio smaller than 0.9.
 10. The multi-apertureimaging system of claim 9, wherein the second camera has an optical pathof less than 9 mm.
 11. The multi-aperture imaging system of claim 10,wherein the second camera has a fixed focal length.
 12. Themulti-aperture imaging system of claim 11, wherein the first camera hasa fixed focal length.
 13. The multi-aperture imaging system of claim 10,wherein the multi-aperture imaging system has a still mode and a videomode.
 14. The multi-aperture imaging system of claim 9, furthercomprising a camera controller connected to the first camera and to thesecond camera, the camera controller configured to output a fused imageusing the first image and the second image.
 15. The multi-apertureimaging system of claim 14, wherein the fused image is generated byregistering pixels within the second sensor to pixels within the firstsensor.
 16. The multi-aperture imaging system of claim 14, wherein thefused image is generated by registering pixels within the first sensorto pixels within the second sensor.